GOTO is a vendor independent international software development conference with more that 90 top speaker and 1300 attendees. The conference cover topics such as .Net, Java, Open Source, Agile, Architecture and Design, Web, Cloud, New Languages and Processes

Presentation: "The Evolution of Hadoop at Spotify - Through Failures and Pain"

Track: The State of Data / Time: Tuesday 17:00 - 17:50 / Location: Grandball

The quickest way to learn and evolve infrastructure is by encountering obstacles and being forced to overcome limitations that keep you inches away from project goals. At Spotify, we've encountered many of these obstacles and frustrations as we grew our Hadoop cluster from a few machines in an office closet aggregating played song events for financial reports, to our current 900 node cluster that plays a large role in many features that you see in our application today.

Two members of Spotify's Hadoop 'squad' will weave in war stories, failures, frustrations and lessons learned to describe the Hadoop/Big Data architecture at Spotify and talk about how that architecture has evolved.

We'll talk about how and why we use a number of tools, including Apache Falcon and Apache Bigtop to test changes; Apache Crunch, Scalding and Hive w/ Tez to build features and provide analytics; and Snakebite and Luigi, two in-house tools created to overcome common frustrations.

Download slides

Josh Baer, Hadoop Product Owner at Spotify

Josh Baer

Biography: Josh Baer

Josh spent six years as a software engineer building infrastructure components at AT&T before discovering the world of 'Big Data' in a class at NYU by O’Reilly author Foster Provost. He 'joined the band' at Spotify in early 2013 and has worked on a small team focusing on stabilizing and enhancing the Hadoop infrastructure, performing multiple migrations, upgrades and growing the cluster from 190 nodes to over 1300. Today, Josh lives in Stockholm, Sweden and leads the platform vision as the Hadoop Product Owner.

Josh holds a BS in Computer Science/Philosophy from the University of Pittsburgh and a MS in Computer Science from NYU.

Twitter: @L_phant

Rafal Wojdyla, Engineer at Spotify

Rafal Wojdyla

Biography: Rafal Wojdyla

Rafal is an engineer at Spotify, a member of Hadoop squad responsible for operating, maintaing and growing one of the biggest Hadoop cluster in Europe. He is a core committer to snakebite – super fast pure python HDFS client.

Twitter: @ravwojdyla